Unusual event detection in wide-angle video (based on moving object trajectories)

ABSTRACT

Object images captured by a wide-angle camera are distorted due to the optical effects of the wide-angle lens. The disclosed innovations allow an automatic analysis on the corrected image distinguishing normal movement from an unusual event movement. The analysis is based on Markov Modeling on moving object trajectories and motion angles.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority from provisional patent application61/097,915, filed on Sep. 18, 2008, which is hereby incorporated byreference.

BACKGROUND

The present inventions relate generally to image processing in awide-angle video camera, and more specifically to tracking movingregions and detecting unusual motion activity within the field of viewof the camera.

Note that the points discussed below may reflect the hindsight gainedfrom the disclosed inventions, and are not necessarily admitted to beprior art.

Video surveillance systems using wide-angle optical systems apply imagedata processing to enhance or obscure visual information as necessaryusing program algorithms. It is useful to be able to define the extentswithin which image processing operations should take place; for examplemoving region detection and object tracking software may define thebounding box of a suspicious object and use it to direct enhancementprocessing to the appropriate area of the image. Regions of interest(ROIs) can be identified in the image based on motion, color, behavior,or object identification. Computer algorithms and programs can beimplemented at the camera or at a monitoring base station to automateselection of the ROIs, motion tracking, sound an alarm, summon emergencyresponders, activate responsive systems (e.g., close doors, sealbulkhead, lock doors, activate ventilation or fire-suppression system,activate lighting, etc.), or other actions as programmed into thesystem. Further, the object tracking, or alarm, or other processing canbe performed on the corrected or on uncorrected versions of the imagesor video.

In many surveillance systems, standard pan-tilt-zoom (PTZ) camerascapture images of an area. A PTZ camera relies on mechanical gears toadjust the pan, tilt, and zoom of the camera. These cameras have variousdisadvantages or limitations in surveillance system and applications.Typically, adjusting to concentrate on a region of interest (ROI)necessarily requires a PTZ camera to adjust its view to the ROI and losesight of the overall area. PTZ cameras are also prone to mechanicalfailure, misalignment during maintenance, and are relatively heavy andexpensive.

In more recent surveillance systems, a new class of cameras replaces themechanical PTZ mechanisms with a wide-angle optical system and imageprocessing circuitry and software. This type of camera monitors awide-angle field of view and selects ROIs from the view to transmit to abase station; in this way it emulates the behavior of a mechanical PTZcamera. The wide-angle view can be as much as 180° and capture a fullhemisphere of visual data. The wide-angle optics (such as a fisheyelens) introduce distortion into the captured image and processingalgorithms are used to correct the distortion and convert it to a viewthat has a similar view and function as a mechanical PTZ camera. Themovements and zoom function are emulated by image data processingaccomplished by internal circuitry.

However, these innovations can be used in a camera with a view angleconsiderably less than 180°. The inventions can be used with annularlenses that often provide imagery that is not from 0 to 90 degree in thetilt (phi in spherical coordinates), but perhaps 40 to 130 degrees.Another lens type can use an ovalized rectangle shape. References towide-angle cameras include lens systems that meet these criteria.

The captured 3D world space hemispherical image from a wide-angle cameraappears distorted, and it can comprise what is sometimes referred to asa fisheye view. Image processing algorithms can correct this view to amore conventional 2D depiction on a monitor and also emulate themovement of a PTZ camera. The wide-view camera can simultaneously focuson multiple ROIs while still able to monitor the area within its entirefield of view. This type of camera may be used to improve the processingof selected ROIs, because a wide field is always captured by the camera,and there is no need for mechanical movement and adjustment of thecamera's lens system.

The image processing is typically performed on digital image data. Thecaptured image data is converted into a digital format for image dataprocessing at the camera. The processed data can than be transmitted toa base station for viewing. The image data can be converted to an analogsignal for transmission to the base station, or the image data can beleft in the processed digital format. Additionally, the input signal tothe base station, either analog or digital, can be further processed bythe base station. Again, image processing and ROI selection at thecamera can be fully automatic or subject to various control inputs oroverrides that effect automatic image processing. Manual, automatic, ora combination of control options utilizing the combination of the basestation and internal camera circuitry can be implemented.

Wide-angle surveillance is necessarily implemented in many CCTVapplications. Cameras such as dome cameras and cameras with fisheye orperipheral lenses can produce wide-angle video. A major weakness ofwide-angle surveillance cameras and systems is that they do not have thecapability to produce automatic alarms to unusual events due to movingobjects within the viewing range of the camera.

SUMMARY

The present application discloses new approaches to correcting theoptical distortion to an image of a rectangular object to reconstructthe object for both camera position distortion and distortion effectsintroduced by the optical system.

In some embodiments, the inventions disclose methods and systems thatinterpret the trajectories of moving objects and people in a wide anglevideo. Moving objects in wide angle video are determined, resulting inblobs of pixel locations corresponding to moving objects. Each blob istracked over time and trajectories consisting of coordinates of blobsare determined. The trajectories are transformed into a corrected imagedomain, and then each trajectory is fed to a set of Markov Models (MM),which are trained with prior trajectory data corresponding to regularand unusual motion trajectories of moving objects. Markov Models trainedusing regular motion produce higher probability values for regularmoving objects compared to MMs trained using unusual event trajectories.Whenever there is an unusual motion in wide angle video, the MMcorresponding to unusual events will produce the highest probability.

The disclosed innovations, in various embodiments, provide one or moreof at least the following advantages. However, not all of theseadvantages result from every one of the innovations disclosed, and thislist of advantages does not limit the various claimed inventions.

-   -   Cheaper.    -   Can be implemented at the camera.    -   Can implement with software upgrade.    -   Uses existing wide-angle camera infrastructure.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed inventions will be described with reference to theaccompanying drawings, which show important sample embodiments of theinvention and which are incorporated in the specification hereof byreference, wherein:

FIG. 1 schematically shows an example of an electronic PTZ wide-anglevideo camera.

FIG. 2 schematically shows a cross-sectional view of a video camera witha fisheye lens compatible with the invention.

FIG. 3 is a fisheye image frame of a wide-angle video.

FIG. 4 is the distorted image view of FIG. 3 corrected as a panoramicimage of the room.

FIG. 5 is an exemplary MM structure shown for a regular motion and onefor unusual event motion.

DETAILED DESCRIPTION OF SAMPLE EMBODIMENTS

The numerous innovative teachings of the present application will bedescribed with particular reference to the presently preferredembodiment (by way of example, and not of limitation).

FIG. 1 shows one example implementation of a preferred embodiment of anelectronic PTZ camera, this example being described in terms of a videocamera 100. The camera 100 includes an optical system 102 thatcommunicates with image sensor 104. In some embodiments, the opticalsystem 102 is a wide angle, anamorphic, annular, or panoramic lenssystem (for example), possibly including multiple lenses, holographicelements, or other elements. The lens is typically configured to coveran approximate 180° field. Sensor 104 passes a captured image to imageprocessing circuitry 106 for processing, such as correction of anydistortions of the image caused by the optical system 102 (though otherprocessing is also possible). Examples of image processing that might beperformed include geometric transformations (zoom-in, zoom-out, rotate,etc.), color correction, brightness and contrast adjustment, shading,compositing, interpolation, demosaicing, image editing, or segmenting.PTZ emulating image processing occurs in the image processing circuitry106. Other processes performed on the image data are possible. In someembodiments, the optical system distortion is corrected though the useof tables of correction values that define the necessary warping, suchas that caused by the lens system or other known factors.

The corrected image is preferably compressed and formatted by circuitry108 before being output, such as to a base station (not shown) formonitoring. The camera 100 is preferably controlled by the base stationthrough control circuitry 110. It is noted that other architectures andcamera arrangements can implement the present innovations, and theexample of FIG. 1 is only illustrative and not limiting.

The control circuitry 110 is used to control aspects of operation of theimage processing circuitry 106. Various control options can beimplemented for circuitry 110 and 106. Operation can be configured forautomatic control selecting ROIs based on operating algorithms such asface recognition, motion detecting, event detection, or otherimplemented parameters. PTZ control can operate to focus on one or moreROI simultaneously while maintaining surveillance throughout the fieldof view. Control from a base station can be automatic or manual, withthe ability to manually select ROIs, adjust image quality parameters, orotherwise adjust and manipulate the image or PTZ view. Control canessentially be automatic within the camera with manual adjustments oroverrides performed by the base station. This flexibility to control andvary the captured views by data processing can be thought of asimplementing one or more virtual cameras, each able to be independentlycontrolled, by processing captured image data from the single opticalsystem, or even a combination of several optical systems, and emulateone or more PTZ cameras.

Various optical distortion effects can be introduced by the opticalsystem 102, i.e. lens configuration that the image processing circuitry106 can compensate. For example, in geometric optics and cathode raytube (CRT) displays, distortion is a deviation from rectilinearprojection, that is, a projection in which straight lines in a scenefail to remain straight in an image. This is a form of opticalaberration. Although distortion can be irregular or follow manypatterns, the most commonly encountered distortions are radiallysymmetric, or approximately so, arising from the symmetry of aphotographic lens.

Radial distortion can usually be classified as one of two main types: 1)barrel distortion and 2) pincushion distortion. In “barrel distortion”,image magnification decreases with distance from the optical axis. Theapparent effect is that of an image which has been mapped around asphere. Fisheye and similar lenses, which take hemispherical views,utilize this type of distortion as a way to map an infinitely wideobject plane into a finite image area. In “pincushion distortion”, imagemagnification increases with the distance from the optical axis. Thevisible effect is that lines that do not go through the centre of theimage are bowed inwards, towards the centre of the image.

Radial distortion is a failure of a lens to be rectilinear: a failure toimage lines into lines. If a photograph is not taken straight-on then,even with a perfect rectilinear lens, rectangles will appear astrapezoids: lines are imaged as lines, but the angles between them arenot preserved (tilt is not a conformal map). This effect can becontrolled by using a perspective control lens, or corrected inpost-processing, such as image data processing.

Due to perspective, cameras image a cube as a square frustum (atruncated pyramid, with trapezoidal sides)—the far end is smaller thanthe near end. This creates perspective, and the rate at which thisscaling happens (how quickly more distant objects shrink) creates asense of a scene being deep or shallow. This cannot be changed orcorrected by a simple transform of the resulting image, because itrequires 3D information, namely the depth of objects in the scene. Thiseffect is known as perspective distortion. This radial distortion can becorrected by algorithms operating in the camera.

The described embodiments include the capability to select a region of acaptured image (such as one or more frames of video), whether processedor not, and to perform other data processing on that region. In oneexample implementation, the innovative camera captures an image, such aswide angle video (but not limited thereto), and corrects the wide angleview to create a corrected view (i.e., not distorted or less distortedview) that is sent to an operator. The operator (or specially designedsoftware) can then define or designate a ROI for observation. In otherembodiments, some processing is performed at different steps, such asobject tracking, behavior analysis, motion detection, objectrecognition, or face recognition (for examples). These and otherexamples are described more fully below.

Recent advances in wide angle camera technology [e.g., US Patent Pub.2007/0124783, U.S. Pat. No. 7,366,359, U.S. Pat. App 11/184,720, U.S.Pat. No. 5,684,937] and algorithmic developments in object tracking[U.S. Pat. No. 6,590,999, U.S. Pat. No. 6,394,557, public domaindocument C. Wren, A. Azarbayejani, T. Darrell, and Alex Pentland,“Pfinder: Real-time tracking of the human body,” IEEE Transactions onPattern Analysis and Machine Intelligence, 19(7):780-785, 1997] make itfeasible to extract a complete trajectory of an object from the wideangle video in a room or in a monitored scene. It is then possible tointerpret the intention of the moving object based on this trajectory.In this application, a method and a system will be described fordetecting unusual activity based on moving object trajectories.

The first step of most computer vision movement tracking algorithms isthe segmentation of foreground objects from the background. In theseapproaches, foreground objects are moving objects. Moving objects inwide angle video can be determined by the background subtraction method.There are several background estimation methods listed in the literature[e.g., U.S. patent application Ser. No. 11/203,807; Stauffer, C,Grimson, “Adaptive background mixture models for real-time tracking,”In: Proc. IEEE Computer Society Conf. on Computer Vision and PatternRecognition, Vol. 2, pp. 246-252, 1999; and R. Collins, A. Lipton and T.Kanade, “A System for Video Surveillance and Monitoring,” in Proc.American Nuclear Society (ANS) Eighth International Topical Meeting onRobotics and Remote Systems, Pittsburgh, Pa., Apr. 25-29, 1999]. Inthese prior art systems, a so-called background image is obtained fromthe video by either recursively averaging the past images of the videoor by median filtering.

Since a moving object temporarily enters and leaves the image, thecontribution of moving object pixels are immaterial compared to pixelscorresponding to background objects when the video image frames areaveraged over long time intervals, or the median value of a given pixelis probably due to the background of the scene, because most of thepixel values captured by the sensor should be due to static backgroundobjects. The estimated background image is subtracted from the currentimage of the wide angle video to determine moving pixels. As a result ofthis operation, blobs of remaining pixels in the image correspond tomoving objects.

The system can also have a computer program comprising amachine-readable medium having computer executable program instructionsthereon for executing the moving object detection and object trackingalgorithms fully in the programmable camera device, such as described inU.S. patent application Ser. No. 10/924,279, entitled “Tracking MovingObjects in Video Using Wavelet Domain Information,” by A. E. Cetin andY. Ahiska, which is hereby incorporated by reference.

The moving blob of pixels in the i-th image frame of the wide anglevideo is associated with the corresponding blob in the i+1-st imageframe of the wide angle video using an object tracking method (e.g., asdescribed in patent application Ser. No. 10/924,279 in which themean-shift tracking method used is based on the wavelet transformcoefficients of the moving blob of pixels). The various optionsavailable can include analysis tracking motion or movement. In oneembodiment, a plurality of video frames is compressed using a wavelettransform (WT) based coder, and the trajectory of moving objects isdetermined based on the histogram of the wavelet coefficients of themoving objects. This tracking is preferably carried out by mean-shiftanalysis of the histogram formed from the wavelet domain data of videoframes. Another option compresses the video using a WT based coderwithout completely reconstructing the original video data to obtainwavelet data. Movement detection can operate on actual or compressedvideo data. Because wavelet coefficients carry both space and frequencyinformation about an object, a histogram constructed from wavelettransform coefficients also contains structure information. The use ofwavelet domain color information leads to a robust moving objecttracking system. Other object tracking methods listed in the literaturecan be also used for tracking moving objects in wide angle video.

For example, let the center of mass of the blob of moving object pixelsbe x_(i) at a given time instant. When the next image frame of the videoarrives, this image is also subtracted from the background image and anew blob of moving object pixels is estimated. The center of mass thenew blob x_(1+i) will be different from the vector x_(i) because theblob is due to a moving object. In this way, a sequence of vectors x₁,x₂, . . . , x_(N) representing the trajectory of the moving object isdetermined from the video.

The trajectory of an object in a straight path will be different in awide angle camera compared to a narrow angle camera. In a narrow anglecamera, the trajectory will be almost linear or the difference vectorswill be almost the same for an object with a constant speed. In a wideangle camera on the hand, the trajectory will be curved and unevenbecause of the distorted nature of the image in a wide angle camera.Therefore, the trajectory interpretation methods [see e.g., FatihPorikli, “Trajectory distance metric using hidden markov model basedrepresentation,” IEEE European Conference on Computer Vision, PETSWorkshop, 2004] developed for narrow angle cameras cannot be applied towide angle cameras.

The disclosed embodiments herein describe an innovative event detectionmethod for wide angle video. In this approach, the trajectory of amoving object is corrected and the corrected trajectory fed to a bank ofMarkov Models (MM) with well-defined states for interpretation. Markovmodels with designer-defined states have not been used in video dataprocessing. Hidden Markov Models are the most widely used recognitionengines for voice recognition, they are also recently used in image dataprocessing, but in HMMs the states are determined according to analgorithm described in Lawrence Rabiner, Biing-Hwang Juang, Fundamentalsof Speech Recognition, February 1993, Prentice Hall-NY. This increasescomputational cost significantly compared to Markov models with userdefined states, because there is no need to estimate the hidden states.We do not use “hidden” states in Markov models in this process leadingto computationally more efficient implementations, because there is noneed to compute the probabilities due to hidden states in Hidden MarkovModels (HMM). By defining the states of Markov models manually, we avoidthe state estimation step of HMM algorithms. In speech processing it isnot possible to define the states manually, however, we define states ofMarkov models according to the motion vector angles of moving objects inthis approach.

FIG. 2 schematically shows a cross-sectional view of a video camera witha fisheye lens compatible with the invention. When a conventionalfisheye lens 205 with a 180° field of view is installed into a videocamera 210, image frames of the video produced by the camera aredistorted (see FIG. 4). A sensor plane 215 receives light rays 220entering through the lens 205. The human FIG. 225 and other objectscaptured by the camera 210, especially on the periphery of the 180°view, appear distorted. This distortion is typically circular for acircular fisheye imaging system, but can be of other shapes, dependingon the lens system installed.

Systems and methods for transforming a wide-angle image from oneperspective form to another have been implemented using differenttechniques, and generally may be divided into three separate categories:

-   -   a) tabular distortion-correction systems and methods;    -   b) three-dimensional (3D) projection systems and methods; and    -   c) two-dimensional (2D) transform mapping systems and methods.

The first category includes U.S. patent application Ser. No. 10/837,012,entitled “Correction of Optical Distortion by Image Processing,” whichis hereby incorporated by reference. The distortion is corrected byreference to a stored table that indicates the mapping between pixels ofthe distorted image and pixels on the corrected image. The table istypically one of two types: 1) a forward table in which the mapping fromdistorted image to corrected image is held, or 2) a reverse tableholding the mapping from corrected image to distorted image. On theother hand, U.S. patent application Ser. No. 10/186,915, entitled“Real-Time Wide-Angle Image Correction System and Method for ComputerImage Viewing,” which is hereby incorporated by reference, generateswarp tables from pixel coordinates of a wide-angle image and applies thewarp table to create a corrected image. The corrections are performedusing a parametric class of warping functions that include SpatiallyVarying Uniform (SVU) functions.

The second category of systems and methods use 3D computer graphicstechniques to alleviate the distortion. For example, U.S. Pat. No.6,243,099, entitled “Method for Interactive Viewing Full-Surround ImageData and Apparatus Therefor,” which is hereby incorporated by reference,discloses a method of projecting a full-surround image onto a surface.The full-surround image data is texture-mapped onto a computer graphicsrepresentation of a surface to model the visible world. A portion ofthis visible world is projected onto a plane to achieve one of a varietyof perspectives. Stereographic projection is implemented by using aspherical surface and one-to-one projecting each point on the sphere topoints on an infinite plane by rays from a point antipodal to the sphereand the plane's intersection.

The third category includes U.S. Pat. No. Re 36,207, entitled “OmniviewMotionless Camera Orientation System,” which is hereby incorporated byreference, discloses a system and method of perspective correcting viewsfrom a hemispherical image using 2D transform mapping. The correction isachieved by an image-processor implementing an orthogonal set oftransform algorithms. The transformation is predictable and based onlens characteristics.

In order to interpret the motion of a moving object, it is enough tocorrect the indices of moving blobs in the video for motion basedunusual event detection. Because the unusual event detection method isbased on the trajectories of moving objects in the viewing range of thecamera, one can simply correct the trajectory of a moving object beforemaking a decision according to a look up table or a function. Considerthe example of a person walking with a constant speed along a straightline, which does not produce uniformly spaced center of mass locationsin a wide angle video, i.e., |x_(1+i)−x_(i)| will not be the same forall i values even for someone walking just below the camera because ofthe circular nature of the fish-eye image. Although the trajectory willbe a straight line going through the center of the circular image,distance between the two consecutive moving blob center of masses|x_(1+i)−x_(i)| will be smaller when the person is away from the camera210 compared to those distance values beneath the camera.

Rabiner and Juang describe three problems that need to be addressed inHidden Markov Models:

-   -   1. Compute probability of the event given a set of feature        parameters describing the event.    -   2. State estimation for 1 because states are “hidden.”    -   3. Training for 2 and state transition probabilities.        The unusual event detection therefore depends on a state        estimation probability that an object vector corresponds to        unusual as opposed to normal behavior or movement.

In FIG. 3, a fisheye image frame of the wide-angle video is shown. Amoving object 305 (i.e., a standing person in the room) moves towardsthe right of the room. A motion vector 310 is also shown in the image(although the person moves to the right of the room the motion vectorpoints downwards in the fisheye image). This fisheye distorted imageview is corrected in FIG. 4 as a panoramic image of the room. Thecorrected motion vector 410 is also shown in FIG. 4, and the motionvector now points towards the right.

Trajectory correction can be made by using a look up table describing afunctiony=g(x)where x represents coordinates (or indices of pixels) in the fisheyeimage and y represents the coordinates in the corrected domain (e.g.,Cartesian coordinates). For example, the standing person appearinghorizontal in the fisheye image (FIG. 3) is corrected in FIG. 4 underthe transformation function g. The function g changes according to thenature of the lens but it is not difficult to generate a look up tablesummarizing it for a given wide angle lens. A corrected version of theimage shown in FIG. 3 is shown in FIG. 4. Thus, we track movement in anon-rectilinear image.Unusual Event Detection Using Corrected Moving Object Trajectories

In this section, we describe how we process the corrected locations ofmoving objects for unusual event detection. We define the stateaccording to the angle of the vector in the image plane, and we willdefine several scenarios for unusual events based on moving objecttrajectories.

Scenario 1:

y₁, y₂, y₃ . . . , y_(N+1) are the corrected locations of a movingobject at each image frame of a wide angle video. Obtaining a set ofdifference vectorsv ₁ =y ₂ −y ₁ ,v ₂ =y ₃ −y ₂ , . . . ,v _(N) =y _(N+1) −y _(N)The vectors v₁, v₂, . . . , v_(N) are motion vectors, and most of themotion vectors are almost equal to each other for a regular movingobject or a person moving in a room. A regular person knows where to go,e.g., he enters the room and goes to his desk. In contrast, in anunusual event the vectors will not be equal to each other. Because therewill be sudden turns, irregular movement, searching for something in theroom, moving up and down, and visiting/searching multiple desks in aroom, etc. Accordingly, the angle of the vector varies irregularly.Additionally, the magnitude of the vector (e.g., running, rapid headmovement, etc) can be used for some screening to indicate an unusualevent.

The operator may also define unusual events. For example, if someoneexits from a door in the wrong direction or an area in the viewing rangeof the camera may be a prohibited zone etc. These will be the subject ofScenario 2.

Smoothing the difference vectors using a low-pass filter withcoefficients (¼, ½, ¼) and obtaining another set of vectors:W=w[1],w[2], . . . w[N]After this step, calculating the magnitude and phase angles with respectto a reference line of each vector to convert the Cartesian motionvectors into polar coordinates, i.e.,w[n]=w _(n)exp(θ_(n))where w_(n) is the magnitude of the w[n] vector and the θ_(n) is thephase angle with respect to a reference line.Markov Models (MM) Based on Motion Angles:

Consider the Markov Model (MM), a type of stochastic model, structureshown in FIG. 5. Each state is defined according to the angle of themotion vector. State 1 corresponds to 0 to theta (θ) degrees withrespect to a reference line, State 2 covers theta to 2θ, etc and State 3corresponds to 2pi-theta (2Πθ) to 2Π. In each state, the observationvariable is the magnitude of the motion vector. The magnitude of themotion vector can be characterized by a mixture of Gaussian probabilitydensity functions (pdf). One can even use a single Gaussian with mean,m, and standard deviation sigma to simplify computations. Since thestates of Hidden Markov Models are not defined by the designer or theuser in HMMs, they have to be determined using a computationally complexstate estimation algorithm. In this implementation, states of MMs aremanually defined according to the motion vector angles of the movingobjects, simplifying the computations involved. The state estimationproblem is straightforward in MMs. They are simply determined accordingto the angles of motion vectors.

Two non-hidden Markov Models with three states S1, S2, and S3 eachcovering θ=120 degrees are shown in FIG. 5. The transition probabilitiesare defined as follows: a_(ij)=Prob(State j at t=n+1|State i at t=n).

Markov Model A is for regular events and Markov Model B is for unusualevents. Each Markov model has the same structure but the transitionprobabilities differ for the model describing the regular events andunusual events, respectively. In regular motion, phase angles of w[n]remain the same and do not change very often compared to unusual eventsin which the subject may change direction suddenly or even zig-zag in aroom. Therefore, probabilities a_(ii) will be larger than a_(ij), i notequal j. Also, the magnitudes of the motion vectors do not typicallychange during regular motion, which may corresponds to a person walkingwith a steady pace. On the other hand, the magnitude is expected tochange in an unusual event, and as noted above, may be higher in anunusual event scenario (e.g., running, rapid head movement, rapid armmovement, etc).

The recognition problem is to find the model producing a givenw[n]=w_(n) exp(θ_(n)) sequence with highest probability. The first stepof the MM based analysis (in both the recognition and the trainingphases) consists of estimating a state transition sequence from themoving object data. This is done according to the motion vector angles.An example state transition sequence may look like:C=(S1,S1,S2,S2,S1, . . . ,S3,S1)At each state Si, we observe a motion vector magnitude w_(i). Theprobability of obtaining the transition sequence C and the motion vectormagnitudes w_(i) in Model A is equal toProb(W|A)=p _(S1) *a ₁₁ p ₁ *a ₁₂ *p ₂ *a ₂₂ *P ₃ *a ₂₁ *p ₄* . . . *a₃₁where p_(S1) is the probability of starting with state S1 and theobservation probability pi=Prob(w₁) is the probability of observing themotion vector magnitude w₁ in state S1, p2=Prob(w₂), etc. which areestimated using typical regular motion events recorded a priori. Thisprocess is called the training phase of MM. Clearly, the probability ofobtaining the transition sequence C and the corresponding motion vectormagnitudes for Model B is equal toProb(W|B)=p _(s1) *b ₁₁ *p ₁ *b ₁₂ *p ₂ *b ₂₂ *P ₃ b ₂₁ *p ₄ * . . . *b₃₁During the recognition phase, both of the above conditionalprobabilities are computed for a given sequence C, and the modelproducing the highest probability is selected. Model probabilities areestimated from typical unusual event videos recorded a priori. Fastrecursive algorithms for computing Prob(W|A) and Prob(W|B) usinglogarithmic look-up tables are described in the book by Rabiner andJuang.

More transitions should occur between states when monitoring an unusualevent compared to regular human motion or regular motion of movingobjects. Hence, the probabilities of transitions between differentstates, a_(ij)'s, i≠j are higher than in-state transition probabilities,a_(ii)'s, for the regular motion model. On the other hand, thetransition probabilities b_(ij), i≠j values will be relatively higher inan unusual event. Therefore, in an unusual event—Prob(W|B)>Prob(W|A)There can be more than one Markov Model (MM) describing regular events.In the case of an unusual event, Prob(W|B) is higher than theconditional probabilities of all MMs representing regular events. Whenthere is more than one MMs for unusual events, it is sufficient to havethe conditional probability of one of these to be higher than theconditional probability of regular MMs to detect an unusual event.Examples of possible MM for unusual events include one for hiding, onefor rifling, one for accident, one for falling, one for forbidden areas,etc.Scenario 2:

In a hallway or corridor, people may be expected to walk only in onedirection. If someone stands still, loiters, or walks in the oppositedirection, then the wide-angle camera equipped with a MM based motionanalysis software produces an alarm. Trajectories of moving objects canbe modeled as a classification problem.

Class 1: People moving in the correct direction. A Markov Model (MM) isdesigned to represent the motion of people moving in the correctdirection.

Class 2: People moving in the opposite direction or standing still orzig-zaging in the hallway. Two Markov Models are designed for thisclass. One MM is allocated to people zig-zaging or meandering people inthe room or hallway. The second MM represents people who move too slow.A third MM is designed for people standing still in the room. In thiscase, the person moves a little bit in the room and stops somewhere inthe middle. This set of MMs can also be used for detecting suddenlystumbling and falling people in a room or a hallway or someone exitingfrom a forbidden exit.

In the next section we describe the design (or training) problem of MMs.

Training of MMs:

The training process of MMs consists of estimation of transitionprobabilities and observation probabilities of each Markov Model. Thisis achieved with the help of recorded videos containing both regular andunusual events. Motion vector sequences w[n] are estimated from eachvideo. Transition probabilities and observation probabilities of MMsrepresenting usual events are estimated from the motion vector sequencesof regular events. Similarly, transition probabilities and observationprobabilities are estimated from the motion vectors of unusual eventvideos. The probability estimation procedure is discussed in the book byL. Rabiner and B-H. Juang.

Scenario 3:

Location normally is not used for normal behavior, but it can be usedfor unusual event detection. Assume that the operator marks a region inthe viewing range of the wide angle camera as a forbidden zone. It ismore natural to select the forbidden zone in the corrected image domain.Some wide angle cameras have the built-in feature of providing correctedimages to the users as described by the U.S. Pat. No. 7,366,359 issuedto Grandeye Ltd. and incorporated by reference. In such systems themarked region in the corrected image domain is mapped to the fisheyeimage, and it may be sufficient to check the coordinates of center ofmass, x₁, x₂, x₃, . . . , x_(N+1), of moving objects to produce analarm. Whenever one of the center of mass coordinates is in theforbidden region an alarm is generated.

The operator may wish to allow people to stay in a forbidden zone for ashort pre-specified duration similar to the 3 second zone in basketball.In this case, one needs to monitor the duration of center of masslocations. An alarm is issued whenever a subset of center of masslocations x_(i), x_(i+1), x_(i+2), . . . , x_(i+L), stays inside theforbidden zone.

It should be clear to those skilled in the art that the techniquesdisclosed above might be applied to more than one region within acamera. The foregoing has described methods for the implementation ofimage processing within regions that are given for illustration and notfor limitation. Thus the invention is to be limited only by the appendedclaims.

According to a disclosed class of innovative embodiments, there isprovided: A method of monitoring moving objects in a wide-angle video,comprising the steps of: determining moving object trajectories;converting the moving object trajectories to a trajectory in aperspectively corrected image domain; and interpreting the objecttrajectories for detecting unusual behavior using state transitionprobability models of non-hidden Markov models.

According to a disclosed class of innovative embodiments, there isprovided: A method of monitoring moving objects in a wide-angle video,comprising the steps of: determining moving object trajectories; andinterpreting the object trajectories for detecting unusual behavior foran unusual event in a distorted image domain using a state transitionanalysis of Markov Models.

According to a disclosed class of innovative embodiments, there isprovided: A method for designating a pixel group movement as an unusualevent, comprising the steps of: determining movement trajectories of agroup of pixels corresponding to a moving object; and associating themoving object trajectories with unusual behavior designated as anunusual event in a non-rectilinear image domain using Markov models.

According to a disclosed class of innovative embodiments, there isprovided: A method for classifying movement as an unusual event,comprising the steps of: calculating movement trajectories for a blob ofpixels on a image data set; training a plurality of state transitionprobability models with prior trajectory data corresponding to regularand unusual motion trajectories of moving objects; and analyzing themovement trajectories for the blob and assigning an unusual eventclassification according to the outcome of probability calculations onthe state transition probability models.

According to a disclosed class of innovative embodiments, there isprovided: A system for monitoring moving objects in a wide-angle video,comprising: a wide-angle lens on a digital video camera capturing asequence of video images; and an image data processor detecting andconverting moving object trajectories into vectors in a perspectivelycorrected image domain and interpreting the object vectors as unusualbehavior according to a stochastic model, which is not a hidden MarkovModel, in which the states are defined according to motion vector anglesof moving objects.

According to a disclosed class of innovative embodiments, there isprovided: A system for monitoring moving objects in a wide-angle video,comprising: a wide-angle lens on a digital video camera capturing asequence of video images; and an image data processor detecting andconverting moving object trajectories into vectors in a perspectivelycorrected image domain and interpreting the object vectors as unusualbehavior according to a stochastic model, which is not a hidden MarkovModel, in which the states are defined according to motion vector anglesof moving objects.

According to a disclosed class of innovative embodiments, there isprovided: A system for detecting unusual events in a wide-angle video,comprising: an electronic camera that emulates PTZ function capturingwide-angle digital video images; and an image data processor thatcalculates and converts moving pixel blob trajectories in anon-rectilinear image domain and analyzes the blob trajectories asunusual behavior using state transition probability models applied to adata object histogram; wherein the state transition probability modelsare trained using prior trajectory data corresponding to regular andunusual motion trajectories of moving objects.

According to a disclosed class of innovative embodiments, there isprovided: A system for classifying a movement as an unusual event in awide-angle video, comprising: a digital PTZ camera capturing distortedimages and converting the images into a video stream of non-rectilinearvideo images; image data processing circuitry operating analysissoftware algorithms that segment moving object pixels from stationarypixels, determine corrected motion vectors for the moving object,analyze the vectors according to Markov models, and classify themovement as normal or unusual; wherein the movement is classifiedaccording to the conditional probability if ones of unusual event Markovmodels exceed the conditional probability of ones of regular eventMarkov models.

Modifications and Variations

As will be recognized by those skilled in the art, the innovativeconcepts described in the present application can be modified and variedover a tremendous range of applications, and accordingly the scope ofpatented subject matter is not limited by any of the specific exemplaryteachings given.

The references herein to video and still images is not limited to analogor video alone, and can be either variety or any other format or type ofimaging technology.

The innovations of the present application are preferably implementedusing a wide-angle camera, though any type of wide view camera can beused in implementing the present invention, including other anamorphicand non-anamorphic imaging systems. Additionally, although a videocamera is described a camera capturing still images at a periodic timeinterval can be used.

The various innovations can be implemented via one or more internalcamera image processers or on a separately connected base station, or acombination of the two.

The innovations can also be implemented using multiple cameras invarious configurations. Two cameras can be positioned with overlappingfields-of-view with both operating to detect unusual movement or justone, with results from one shared with the other. Thus, one camera canbe used to trigger ROI selection in a different camera.

There may be a size component to consider as well. For example, anexemplary system may not be desirable that identifies ruffling leaves asan unusual event. Or, an oversize truck on a given road with otherwisenormal movement vectors may need to identified as an unusual event.

Similarly, the innovations can be linked with object recognitionalgorithms to perform selective analysis. For example, the system can beconfigured to analyze humans only, trucks only, everyone but identifiedemployees, etc. The combination can also detect abnormal trajectorieswith recognition of objects, e.g., human lying on the floor, carscolliding, etc.

Although a PTZ electronic camera is described, the camera system canalso include a rotation movement, i.e. a PTZR electronic camera.

Generally, at least two MM will be used, but a single MM can be usedwith threshold vector and/or magnitude values indicating an unusualevent.

As has been mentioned above, the examples given herein are onlyillustrative and are not intended to imply that these are the only waysto implement the present innovations. The order of the actions describedherein, and the locations at which they are performed, can of coursevary within the scope of the present innovations. These innovations arealso applicable for other types of processing aside from thosementioned, beyond object tracking, privacy domains, and alarmtriggering. Outside computer systems, such as servers, can be used forcalculating many of the necessary functions, or the camera itself can beequipped with this capability.

Additional general background, which helps to show variations andimplementations, may be found in the following publications, all ofwhich are hereby incorporated by reference:

The edited book by Sergio A. Velastin and Paolo Remagnino entitled“Intelligent distributed video surveillance systems”, published byInstitution of Electrical Engineers in 2006 fails to provide any unusualevent detection methods (based on moving object trajectories) forwide-angle cameras.

United States Patent Application 20060187305 by Trivedi et al., Aug. 24,2006 entitled “Digital processing of video images” describes an HMMbased video analysis method for warped video images but it fails to usemoving object trajectories in Markov models for unusual eventrecognition. A method based on HMMs is computationally more inefficientthan the present approach using Markov Models with well-defined states,which are defined according to the motion vector angles of the movingobjects.

None of the description in the present application should be read asimplying that any particular element, step, or function is an essentialelement which must be included in the claim scope: THE SCOPE OF PATENTEDSUBJECT MATTER IS DEFINED ONLY BY THE ALLOWED CLAIMS. Moreover, none ofthese claims are intended to invoke paragraph six of 35 USC section 112unless the exact words “means for” are followed by a participle.

1. A method of monitoring moving objects in a wide-angle video,comprising the steps of: determining moving object trajectories;converting the moving object trajectories to a trajectory in aperspectively corrected image domain; and interpreting the objecttrajectories for detecting unusual behavior using state transitionprobability models of non-hidden Markov models.
 2. The method of claim1, wherein the object trajectories are determined on the distorted imagedomain captured by the imaging sensor of the wide-angle camera.
 3. Themethod of claim 1, wherein the said distorted wide-angle video iscorrected inside the camera.
 4. The method of claim 1, wherein themoving objects are determined in the distorted image domain using animage background subtraction method involving a recursive estimation ofthe background image in the distorted image domain.
 5. The method ofclaim 1, wherein the detected moving objects are tracked in distortedimage domain using a mean-shift object tracking method operating in thedistorted image domain.
 6. The method of claim 4, wherein the detectedmoving objects are tracked in distorted image domain using a mean-shiftobject tracking method operating in the distorted image domain.
 7. Themethod of claim 5, wherein the said mean-shift tracking method computesa data object histogram from a real-valued discrete wavelet transform.8. The method of claim 1, wherein the said moving object trajectories incorrected image domain are interpreted using Markov Models with statesdefined according to motion vector angles of moving objects for unusualevent detection.
 9. The method of claim 1, wherein the state transitionprobability models are trained with motion vectors in Cartesiancoordinates and polar coordinates computed in corrected image domain.10. A method of monitoring moving objects in a wide-angle video,comprising the steps of: determining moving object trajectories; andinterpreting the object trajectories for detecting unusual behavior foran unusual event in a distorted image domain using a state transitionanalysis of Markov Models.
 11. The method of claim 10, wherein theobject trajectories are determined on the distorted image domaincaptured by the imaging sensor of the wide-angle camera.
 12. The methodof claim 10, wherein the moving objects are determined in the distortedimage domain using an image background subtraction method involving arecursive estimation of the background image in the distorted imagedomain.
 13. The method of claim 10, further comprising the step of:tracking a detected moving object in a distorted image domain using amean-shift object tracking method operating in the distorted imagedomain.
 14. The method of claim 10, wherein the unusual event isdetected when a moving object enters into a forbidden regionpre-specified by an operator.
 15. The method of claim 10, wherein thesaid forbidden region is specified on the perspectively corrected imagedomain.
 16. A system for detecting unusual events in a wide-angle video,comprising: an electronic camera that emulates PTZ function capturingwide-angle digital video images; and an image data processor thatcalculates and converts moving pixel blob trajectories in anon-rectilinear image domain and analyzes the blob trajectories asunusual behavior using state transition probability models applied to adata object histogram; wherein the state transition probability modelsare trained using prior trajectory data corresponding to regular andunusual motion trajectories of moving objects, and wherein the statetransition probability models are non-hidden Markov models.
 17. Thesystem of claim 16, wherein the data object histogram is obtained fromwavelet coefficients of moving objects.
 18. The system of claim 16,wherein the pixel blobs are derived from image data by segmentingnon-rectilinear image data into moving foreground pixels and stationarybackground pixels, wherein a background image is obtained by eitherrecursively averaging the past images of the digital video or by medianfiltering.
 19. The system of claim 16, further comprising: a systemclassification output of an unusual event according to a conditionalprobability if ones of unusual state transition probability modelsexceed the conditional probability of ones of regular event statetransition probability models.